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COMBINED PI SPLAY -CAMERA FOR AN IMAGE PROCESSI NG SYSTEM 

Field of the Invention 

The present invention relates generally to the field of image 
processing systems, and more particularly to a combined display and 
camera suitable for use in such systems. 

Background of the Invention 

Many conventional image processing systems utilize a camera in 
conjunction with a video display. FIG. 1 shows an example of such 
an arrangement in the form of an image processing system 100 which 
includes a camera 102 and a separate display 104. The system 100 
may be used in a video conferencing application or other suitable 
application in which the camera 102 generates an image of an on- 
site user 106 and the display 104 presents images of one or more 
remote users to the user 106. 

A significant problem that can arise in the system 100 of FIG. 
1 is that it is difficult to configure the camera 102 and display 
104 such that it appears to both the on-site and remote users that 
they are making eye contact via the generated images. This eye 
contact problem is a result of misalignment between the optical 
axis of the camera 102 and a normal to the display 104. In the 
system 100, the camera 102 and display 104 are typically located as 
close together as possible in order to minimize this effect. 
However, in applications such as portable video conferencing units 
and human-machine interfaces, users often locate themselves very 
close to the display, such that the eye contact problem is 
accentuated . 

U.S. Patent No. 5,666,155 entitled w Eye Contact Video 
Telephony 1 ' discloses an approach for addressing the eye contact 
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problem described above. In this approach, a video camera is 
located behind a display screen and generates images of a user 
looking at the screen. Displayed images are directed to the user 
via an array of lenses in the display screen, so as to create an 
'5 illusion of eye contact. Unfortunately, this approach requires 
placement of a video camera at a particular distance behind the 
display screen, which can substantially increase the size of the 
system. In addition, such an approach is unable to avoid certain 
important types of image-related distortion, such as the so-called 
10 "fish eye" distortion which results in a single-camera system when 
O a user is situated close to the display. 

^ Summary of the Invention 

Kj The present invention solves one or more of the above-noted 

£l5 problems of conventional image processing systems by providing a 
combined display- camera in which a number of individual camera 
Z elements are integrated with display elements of a display. 
0=? In accordance with one aspect of the invention, a combined 

£ display- camera for use in an image processing system includes an 
U 20 array of interspersed display elements and camera elements, the 
interspersed elements being arranged substantially in a common 
plane of a flat panel display or other type of display. Each of at 
least a subset of the camera elements has one or more imaging 
angles associated therewith, with the one or more imaging angles 
25 being selected to provide a desired imaging operation for the 
combined di splay- camera . The imaging angles of the camera elements 
can be selected to provide an imaging operation which approximates 
that of a lens -based single-camera system or of a pin-hole camera 

2 
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system. Each of the camera elements may include multiple image 
sensors, such that different imaging angles can be set for the 
different image sensors of a given camera element, and different 
perspectives of a scene can thereby be generated in the image 
"5 processing system. 

In accordance with another aspect of the invention, the above - 
noted imaging angles for a given one of the camera elements 

comprise a set of angles including a horizontal angle a K = tan _I (— ) 

d 

and a vertical angle a = tan _1 (— ) , where x and y denote the 

p, ; d 

ill 0 horizontal and vertical distances from the camera element to the 
hi optical axis of the combined display-camera, and d is the distance 
y from an image plane of the combined display- camera to a desired 
Jf virtual focus point of the combined display- camera . 
" s - Advantageously, the present invention solves the above-noted 

Hi 5 eye contact and "fish eye" distortion problems, and avoids the need 
:V for implementation of a camera adjacent to or behind a display. In 
^ addition, the combined display-camera of the present invention is 
p suitable for use in a wide variety of image processing systems, 
including video-camera-based systems such as video conferencing 
2 0 systems, video surveillance and monitoring systems, and human- 
machine interfaces . 

Brief Description of the Drawings 

FIG . 1 is a simplified system diagram illustrating the 
25 configuration of a conventional image processing system including 
separate display and camera elements. 
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FIG. 2 is a simplified system diagram showing a combined 
display- camera in accordance with an illustrative embodiment of the 
present invention. 

FIGS. 3(a) and 3(b) show examples of different possible 
*5 arrangements of interspersed display elements and camera elements 
in a combined display-camera in accordance with the invention. 

FIG. 4 shows one possible implementation of a camera element 
of a combined display-camera in accordance with the invention. 

FIGS. 5 and 6 illustrate in greater detail the operation of 
10 the combined display-camera of FIG. 2. 
Q FIG. 7 shows an example of an image processing system in which 

the combined display-camera of FIG. 2 may be implemented. 

flj Detailed Description of the Invention 

Jl5 FIG. 2 shows a portion of an image processing system 200 in 

s accordance with an illustrative embodiment of the present 

y~ invention. The system 2 00 includes a combined display-camera 210 
5; which is configured as an array of alternating display elements 212 
O and camera elements 214. Each of the display elements 212 may be 
^2 0 a conventional display element such as a liquid crystal display 
(LCD) element, and may correspond to one or more image pixels. Each 
of the camera elements 214 may be a charge-coupled device (CCD) , 
photosensor or other type of conventional image sensor, or a group 
of multiple such image sensors. In operation, the image sensing 
2 5 outputs of the set of camera elements 214 are combined using 
conventional image signal processing techniques so as to provide a 
desired composite image of a scene. 
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The camera elements 214 each preferably have a narrow field of 
view, and are interspersed with the display elements 212 
substantially in a common plane so as to form a flat panel display. 
The combined display-camera 210 can thus provide the desired 
*5 alignment between the camera optical axis and the display normal, 
as is illustrated in the figure. Although a user 216 may be 
constrained to a position in front of the display, such positioning 
is common and often preferred, particularly in portable video 
conferencing units and human-machine interfaces. 
10 FIGS. 3(a) and 3(b) show examples of different possible 

O arrangements of interspersed display elements 212 and camera 
."r elements 214 in a combined display-camera in accordance with the 
H : invention. These figures show the combined display-camera in the 
m viewing direction of user 216 of FIG. 2. FIG. 3(a) shows a combined 
tl5 display-camera 210' in which there is a one-to-one relationship 
b between the display elements 212 and the camera elements 214. FIG. 

iu 3(b) shows a combined display-camera 210" in which there is a 

^ three- to-one relationship between the display elements 212 and the 
C camera elements 214. The arrangements of FIGS. 3(a) and 3(b) are 
~*2 0 shown by way of example only, and numerous alternative arrangements 
could also be used. 

It will be appreciated by those skilled in the art that the 
combined display-cameras as shown in FIGS. 2, 3(a) and 3(b) will 
generally include additional elements such as support, housing and 
25 wiring elements. These elements are omitted from the figures for 
simplicity and clarity of illustration, and may be configured using 
any of a number of well-known conventional techniques. 
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The array of alternating display elements and camera elements 
in the combined display-camera of the present invention can be 
implemented in a number of different ways. For example, collimated 
plates having small holes which align with underlying photosensors 
-5 may be used, the small holes being configured to allow collection 
of light from a user or object being imaged. 

FIG. 4 illustrates a camera element 214 configured in the 
manner described above. The camera element 214 includes a 
photosensor 217 and portions of a pair of collimated plates 218-1 
10 and 218-2. The portions of the plates 218-1 and 218-2 shown in the 
O figure include holes 219-1 and 219-2, respectively. The positions 
C5 of the holes 219-1 and 219-2 are selected to provide a particular 
imaging angle for the camera element 214, such that light arriving 
111 in the direction of the imaging angle is incident on the 
jj_5 photosensor 217 as shown. It should be appreciated that different 
e" camera elements may have different imaging angles associated 
L : therewith. 

Ill In other possible implementations, the camera elements 214 may 

3 each include a corresponding microlens system for directing the 
u 20 light onto an associated image sensor. As another example, the 
camera elements 214 and display elements 212 may be constructed in 
the form of a combined element incorporating a semi-transparent 
tilt-mirror, an LCD display element, and an image sensor element. 
Such an arrangement eliminates the need for the above -noted 
25 collimated plates, but introduces additional complexity. Numerous 
other techniques for implementation of the combined display-camera 
will be apparent to those skilled in the art. 
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It is important when implementing the combined display-camera 
that sufficient light be provided for the image sensors of the 

k 

camera elements. The above-noted microlens systems may be used for 
this purpose, although any shadows produced in the display by the 
~5 use of such systems should be minimized. It is also important to 
provide some type of aperture control so as to allow the image 
sensors of the camera elements to operate in different lighting 
conditions. As one possible alternative, the system could be 
restricted to use under specified lighting conditions. 
10 Advantageously, the arrangement of the alternating display 

O elements 212 and camera elements 214 in the combined display-camera 
Ci 210 allows perfect eye contact between images of on-site and remote 
tl : users, since the camera elements are located in substantially the 
111 same plane as the display elements. In addition, the combined 
,£l5 display- camera 210 reduces the required physical depth of the image 
s processing system 2 00, which is particularly important in certain 

y, applications, such as the above-noted portable video conferencing 
:P and human-machine interface applications. Furthermore, the combined 
e display-camera substantially eliminates the previously-described 
""2 0 "fish eye" distortion problem resulting from use of a single wide 
angle camera to image users or other objects which are close to the 
display. Images generated by the combined display-camera 210 are 
well suited for use with motion algorithms such as background 
subtraction, motion field estimation, finger pointing detection, 
25 head tracking, etc. 

FIG. 5 is a diagram illustrating the manner in which the 
combined display-camera 210 of FIG. 2 can approximate the imaging 
performance of a lens-based single-camera system. As shown, the 
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combined display- camera can approximate the single-camera system by 
setting the imaging angles of each individual camera element in the 
combined display-camera. In this figure, the angle of a given 
camera element is specified as a function f(x,y) which when 
•5 utilized in conjunction with similar setting of the other camera 
elements can produce an image at the plane of an imaging element 
22 0 in a manner similar to a single-camera system. It should be 
understood that the element 22 0 is not an actual element of the 
image processing system, but is instead shown only to illustrate an 
10 effective imaging operation provided by the combined action of 
O setting imaging angles for the individual camera elements. 
3 FIG. 6 is a diagram similar to that of FIG. 5 but illustrating 

the manner in which the imaging angles of the individual camera 
elements can be selected to approximate the imaging operation of a 
j:i5 conventional pin-hole camera. In this example, each camera element 
3 is controlled with its own set of angles which includes one 

y, horizontal angle a x and one vertical angle a y . These angles for a 

1^ given one of the camera elements vary as a function f{x,y) of the 
r £ corresponding horizontal distance x and vertical distance y between 
2 0 the given camera element and the optical axis of the combined 
display-camera 210. The above-noted functions for the set of 
camera elements are used to adjust the imaging characteristics of 
the combined display-camera 210. For example, the combined display- 
camera 210 can approximate a pin-hole camera using angles from 

x v 
25 a = tan~ x (— ) and a = tan _1 (— ), where d is the distance from the 
d y d 

image plane to a virtual focus point at pin-hole 225. 
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The imaging angles of the camera elements 214 may be fixed or 
controllable. The term "selecting" as used herein is therefore 
intended to include selection of a fixed imaging angle for a given 
camera element, e.g., during design or manufacture of the combined 
*5 display-camera, or controllable selection during set-up or 
operation of the combined display-camera, as well as other types of 
imaging angle selection. Controllable selection may be implemented 
at least in part under software control. 

An alternative embodiment of the above-described combined 
10 display-camera 210 may associate more than a single image sensor 
O with each of at least a subset of the camera elements. Such an 
Cjj embodiment would allow different imaging angles to be set for the 
Ijr different image sensors of a given camera element, such that 
nj different perspectives of a scene can be generated in the image 
2:15 processing system. In addition, motion algorithms may be applied 
s to the images generated from the different image sensors in order 

L to generate depth information. 

[U FIG. 7 shows an example of an image processing system 2 50 

Q which includes the above -described combined display- camera 210. The 
"2 0 system 250 further includes a controller 260, a processor 270, a 
video signal processor 2 75, a memory 2 80, and one or more 
input/output (I/O) devices 290. In operation, the controller 260 
controls the image angles of the camera elements of the combined 
display-camera 210 in order to provide a desired imaging operation 
25 in the manner previously described, as directed by the processor 
2 70 using one or more software programs that may be stored in the 
memory 280. The video signal processor 275 also under the control 
of the processor 270 collects and processes video output 
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information generated by the camera elements 214 of the combined 
display- camera 210, and delivers video input to the combined 
display- camera 210 for display to a user via the display elements 
212. Video inputs and outputs of the system 250 may be 
*5 communicated via the one or more I/O devices 290 in a conventional 
manner . 

The system 2 50 may be adapted for use in any of a number of 
different image processing applications, including, e.g., video 
conferencing, video surveillance, human-machine interfaces, etc. 
10 More generally, the system 250 can be used in any application that 
O can benefit from the use of the combined display-camera 210 of the 
present invention. 

M= Elements or groups of elements of the system 250 may represent 

«a corresponding elements of an otherwise conventional desktop or 
i: ; 15 portable computer, as well as portions or combinations of these and 
= other processing devices. Moreover, in other embodiments of the 

:"7 invention, some or all of the functions of the controller 260, 
fy processor 2 70, video signal processor 275, memory 280 and/or other 
H: elements of the system 250 may be combined into a single device. 
u 2 0 For example, one or more of the elements of system 250 may be 
implemented as an application specific integrated circuit (ASIC) or 
circuit card to be incorporated into a computer, television, set- 
top box or other processing device. 

The term "processor" as used herein is intended to include a 
2 5 microprocessor, central processing unit (CPU) , microcontroller, 
digital signal processor (DSP) or any other data processing element 
that may be utilized in a given image processing system. In 
addition, it should be noted that the memory 280 may represent an 

10 
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electronic memory, an optical or magnetic disk-based memory, a 
tape-based memory, as well as combinations or portions of these and 
other types of storage devices. 

The above-described embodiments of the invention are intended 
*5 to be illustrative only. For example, the invention can be 
implemented using other types and arrangements of display elements 
and camera elements. The invention can be used in a wide variety 
of image processing systems, including video conferencing systems, 
video surveillance systems, and other camera-based systems. The 
10 invention can also be implemented at least in part in the form of 
one or more software programs which are stored on an electronic, 
magnetic or optical storage medium and executed by a processing 
device, e.g., by the processor 270 of system 250. These and 
numerous other embodiments within the scope of the following claims 
15 will be apparent to those skilled in the art. 
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Claims 

What is claimed is: 

1. An apparatus for use in an image processing system, the 
apparatus comprising: 

*5 a combined display-camera having a plurality of display 

elements and a plurality of camera elements, arranged substantially 
in a common plane with the display elements being interspersed with 
the camera elements, and wherein each of at least a subset of the 
camera elements has one or more imaging angles associated 
10 therewith, the one or more imaging angles being selected to provide 
a desired imaging operation for the combined display-camera. 

2 . The apparatus of claim 1 wherein at least a subset of the 
display elements comprise liquid crystal display elements. 

3 . The apparatus of claim 1 wherein at least a subset of the 
camera elements comprise charge -coupled device image sensors. 

4 . The apparatus of claim 1 wherein at least a subset of the 
H2 0 camera elements comprise photosensors. 

5 . The apparatus of claim 1 wherein a given one of the camera 
elements comprises at least a portion of a pair of collimated 
plates, and wherein an imaging angle is selected for the given 

2 5 camera element by establishing a corresponding positioning of holes 
in the collimated plates. 
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6. The apparatus of claim 1 wherein the combined display- 
camera comprises a flat panel display. 

7. The apparatus of claim 1 wherein at least a subset of the 
*5 plurality of display elements and at least a subset of the 

plurality of camera elements are arranged in an array which 
includes more display elements than camera elements. 

8. The apparatus of claim 1 wherein the one or more imaging 
10 angles are selected to provide an imaging operation for the 

combined display-camera which approximates that of a lens-based 
single -camera system. 

9. The apparatus of claim 1 wherein the one or more imaging 
15 angles are selected to provide an imaging operation for the 

combined display-camera which approximates that of a pin-hole 
camera system. 

10. The apparatus of claim 1 wherein the one or more imaging 
2 0 angles for a given one of the camera elements comprises a set of 

anqles including a horizontal angle a x = tan" 1 !— ) and a vertical 

d 

angle a = tan" L (— ) , where x and y denote the horizontal and 
y d 

vertical distances from the camera element to the optical axis of 
the combined display- camera, and d is the distance from an image 
2 5 plane of the combined display-camera to a desired virtual focus 
point of the combined display- camera . 
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11. The apparatus of claim 1 wherein each of at least a 
subset of the camera elements has a plurality of image sensors 
associated therewith, such that different imaging angles can be set 
for the different image sensors of a given camera element, and 

*5 different perspectives of a scene can be generated in the image 
processing system. 

12 . A method for use in an image processing system, the 
method comprising the steps of: 

10 providing a combined display-camera having a plurality of 

display elements and a plurality of camera elements, arranged 
substantially in a common plane with the display elements being 
interspersed with the camera elements, and wherein each of at least 
a subset of the camera elements has one or more imaging angles 

15 associated therewith; and 

selecting the one or more imaging angles to provide a 
desired imaging operation for the combined display-camera. 

13. An article of manufacture comprising a storage medium for 
2 0 storing one or more programs for use in an image processing system, 

the image processing system including a combined display-camera 
having a plurality of display elements and a plurality of camera 
elements, arranged substantially in a common plane with the display 
elements being interspersed with the camera elements, and wherein 
25 each of at least a subset of the camera elements has one or more 
imaging angles associated therewith, wherein the one or more 
programs when executed by a processor implement the step of 
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selecting the one or more imaging angles to provide a desired 
imaging operation for the combined display-camera. 
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Abstract 

An image processing system includes a combined display-camera 
# having an array of interspersed display elements and camera 
elements, arranged substantially in a common plane of a flat panel 
*5 or other display. Each of at least a subset of the camera elements 
has one or more imaging angles associated therewith, with the one 
or more imaging angles being selected to provide a desired imaging 
operation for the combined display- camera . The imaging angles of 
the camera elements can be selected to provide an imaging operation 
10 which approximates that of a lens-based single-camera system, a 
Q pin-hole camera system or other type of system. Each of the camera 
J J J elements may include multiple image sensors, such that different 
H imaging angles can be set for the different image sensors of a 

m. given camera element, and different perspectives of a scene can be 
J: 15 generated in the image processing system. 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Application of 



Atty. Docket 



ANTONIO J. COLMENAREZ ET AL 



US000320 



o 
i- 




Filed: 



CONCURRENTLY 



Title: COMBINED DI SPLAY -CAMERA FOR AN 



IMAGE PROCESSING SYSTEM 



Commissioner for Patents, Washington, 



D. C . 



20231 



APPOINTMENT OF ASSOCIATES 



Sir : 



The undersigned Attorney of Record hereby revokes all 



prior appointments (if any) of Associate Attorney (s) or Agent (s) in 
the above-capt ioned case and appoints: 



c/o U.S. PHILIPS CORPORATION, Intellectual Property Department, 580 
White Plains Road, Tarrytown, New York 10591, his Associate 
Attorney ( s ) /Agent ( s ) with all the usual powers to prosecute the 
above-identified application and any division or continuation 
thereof, to make alterations and amendments therein, and to 
transact all business in the Patent and Trademark Office connected 
therewith . 



LETTERS PATENT WHEN GRANTED SHOULD BE ADDRESSED TO THE UNDERSIGNED 
ATTORNEY OF RECORD. 



GREGORY L. THORNE 



(Registration No. 39,398) 



ALL CORRESPONDENCE CONCERNING THIS APPLICATION AND THE 



Respectfully, 




Dated at Tarrytown, New York 
on November 18, 2000. 
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